1. Field of the Invention
The invention relates generally to real-time multimedia and more specifically relates to memory management methods and systems used in the rendering and display of real-time multimedia content on computer systems.
2. Discussion of Related Art
There exists a common media form known as “real-time multimedia.” As known in the art, real-time multimedia generally involves the real-time construction, or rendering of graphical imagery on an end-user's computing device, and the subsequent display or presentation of the rendered imagery on a video monitor or viewing device. Computing devices used for executing real-time multimedia presentations may include end-user computers such as personal computers (“PCs”), set-top devices, personal digital assistants (“PDAs”) and workstations (all referred to herein synonymously as “computers”, “personal computers”, “computer systems”, “user systems”, or “PCs”).
In general, a real-time multimedia presentation is provided to the user as a sequence of 2-D images or frames generated from a set of media objects such as 2-D geometry, 3-D geometry, 2-D images, video, and text. The term “real-time” as used herein refers to the fact that a computer program is constructing, or dynamically rendering, a presentation image in time for it to be displayed without the viewer losing a sense of visual continuity. As known in the art, the process of rendering refers to the calculation of pixels from a set of media objects resulting in a 2-D image or frame. The term “visual continuity” refers to the ability to cause the human visual cortex to see a continuous progression of visual events from a time sequence of discrete frames or images that are displayed in quick succession. This technique is used in movie theaters, by displaying a time sequence of pictures at a rate of 24 frames per second. Experts in human vision and signal processing observe that visual continuity decreases as the rate at which a series of pictures is displayed decreases, also known as the “frame rate.” There are many dependent factors that affect visual continuity at a given frame rate, such as the type of multimedia presentation and the activity of the media objects within the presentation, among other factors. Generally speaking, 6 to 7 frames per second may be considered low quality, 8 to 19 frames per second may be considered good quality, and 20 frames per second and above may be considered high quality for multimedia presentations. Visual continuity may be achieved for special purposes in special sequences of images at rates of 5 frames per second or lower. In general, for most common multimedia presentations, visual continuity requires a frame rate of at least 5 frames per second. As defined herein, visual continuity is maintained with a frame rate of 5 frames per second or greater. Therefore the term “real-time multimedia application”, as used herein, refers to a computer program or application that is causing a sequence of images to be rendered and displayed at a frame rate of least 5 frames per second. Real-time multimedia applications are also known herein as “multimedia programs”, “multimedia processes” and “multimedia threads”, and more simply as “applications”, “programs”, “processes”, and “threads”.
Because each frame, or 2-D image, of a real-time multimedia presentation is usually constructed or rendered after the last frame was presented to the viewer, but before the time at which visual continuity would be suspended, input to the computer by a user can be received and affect the course of events in the presentation. Such interaction by the user allows the personal computer to produce an image, or frame, that differs from what would have been constructed and presented had the user not interacted with the presentation. This differs significantly from traditional video, where a series of static, pre-created images are displayed to a viewer in quick succession.
Multimedia applications are usually stored as descriptions that tell the computer how to use various media objects to construct, or render, frames of images through time. Additionally, such descriptions may instruct the computer as to how it should respond to user input during the presentation, allowing for increased utility for the user. Real-time multimedia applications can produce large quantities of visual information from relatively small descriptions based on mathematical and algorithmic descriptions, by combining and rendering media objects in real-time on a viewer's computer.
The data from which real-time multimedia presentation frames are rendered is known herein as “presentation data.” As known in the art, presentation data may include 2-D images, such as textures and video, 2-D and 3-D geometric information, lighting information, blending information, camera information, view port information, implicit position information, implicit rotation information, implicit scaling information, explicit position information, explicit rotation information, explicit scaling information, so called “pixel shader” or high-level graphics instructions, and graphics processor unit (GPU) instructions, as well as other information used for rendering multimedia presentation frames in real-time. The size of the presentation data associated with a single multimedia application varies according to the size and complexity of the 2-D images, 2-D and 3-D geometric information, and other information that define the media objects and other attributes that compose the presentation data.
As the presentation data associated with a multimedia application is rendered into 2-D images, the resulting image data must be stored for subsequent display to a suitable display device. The storage size of the resulting image varies according to the dimensions of the image (measured in pixels) and the number of bytes required to represent each pixel in the image, among other factors. In addition to image data, other important information may be generated and stored during the process of rendering multimedia presentation frames.
Frequently, the amount of memory available to efficiently render a multimedia presentation in real-time is limited. As used herein, memory that is accessible by a graphics processor unit (GPU) for performing operations associated with graphics processing is known as “primary memory”. Primary memory is used to store presentation data for the immediate rendering and storage of frames and associated data. Examples of primary memory may include, but are not limited to, registers or on-chip memory in a separate or integrated GPU, external cache memory associated with a separate or integrated GPU, video memory (VRAM) associated with a separate or integrated GPU, and main system memory (RAM) associated with a separate or integrated GPU. As known in the art, a GPU may be separate from the central processing unit (CPU), typically communicating through an appropriate computer system bus, or integrated partially or fully with a CPU. The memory area used to store multimedia applications with other associated data is known herein as “secondary memory.” Examples of secondary memory may include, but are not limited to, registers or on-chip memory in a CPU, external cache memory associated with a CPU, and main system memory (RAM) associated with a CPU. Primary memory typically provides a computational performance advantage for the task of rendering and displaying multimedia presentation frames over the use of secondary memory, often because of technological or architectural differences within a computer system.
A problem arises when more than one multimedia application attempts to use the same limited, shared primary memory resources. As presently known in the art, the primary memory being used by one or more existing multimedia applications may be completely purged by another multimedia application-suspending the multimedia applications whose presentation data was purged until such time as the offending application exits and the memory becomes available again. Additionally, an application that attempts to allocate primary memory while one or more multimedia applications are active, may be prevented from allocating memory until such time as the memory becomes available or the application requests a smaller portion of memory. Cooperative sharing of primary memory between multimedia applications, either through direct or indirect coordination, does not occur in present multimedia presentation systems. Existing command structures for performing graphics processing such as DirectX and OpenGL are exemplary of such structures presently known in the art that exhibit such problems in the effective sharing of primary memory among multimedia applications.
Such constraints on the allocation and use of primary memory assets for the rendering and display of multimedia presentations are undesirable. Such constraints prevent the efficient, simultaneous use of primary memory by a plurality of multimedia applications in the instance where more primary memory is required than exists within the computer system.
In summary, present media systems are useful for individual real-time multimedia presentations, but fail to provide mechanisms that allow a plurality of real-time multimedia applications to operate concurrently on a computer system with a size-constrained, shared, primary memory pool.